Remote Usability Evaluation at a Glance

Much traditional user interface evaluation is conducted in usability laboratories, where a small number of selected users is directly observed by trained evaluators. However, as the network itself and the remote work setting have become intrinsic parts of usage patterns, evaluators often have limited access to representative users for usability evaluation in the laboratory and the users’ work context is difficult or impossible to reproduce in a laboratory setting. These barriers to usability evaluation led to extending the concept of usability evaluation beyond the laboratory, typically using the network itself as a bridge to take interface evaluation to a broad range of users in their natural work settings. * This paper is based on the third chapter of the thesis work by Castillo (1997). Remote Usability Evaluation at a Glance 2 DEFINITION OF REMOTE USABILITY EVALUATION Remote evaluation is defined as usability evaluation where evaluators are separated in space and/or time from users (Hartson, Castillo, Kelso, Kamler, and Neale, 1996). For consistency of terminology throughout this report, the term remote, used in the context of remote usability evaluation, is relative to the developers and refers to users not at the location of developers. Similarly, the term local refers to location of the developers. Sometimes developers hire outside contractors to do some usability evaluation in a usability laboratory at the contractor’s site. Neither term (local or remote) per the above definitions, applies very well to these third-party consultants, but they (as surrogate developers) could have remote users. In traditional laboratory-based usability evaluation, users are observed directly by evaluators. However, remote and distributed location of users precludes the opportunity for direct observation in usability evaluation. Therefore, with remote evaluation (Figure 1), the network serves as a bridge between users and evaluators, taking interface evaluation to a broad range of networked users (e.g., representative population of users) in their natural work settings. Evaluators User Normal working environment Network Different place and/or time Performs representative and/or everyday tasks Create list of usability problem descriptions Usability laboratory Figure 1. Scenario of a remote usability evaluation session TYPES OF REMOTE EVALUATION METHODS Identifying and distinguishing various approaches to remote evaluation is useful to understand the range of possibilities and to avoid comparison of unlike methods. No claim is made for the completeness of this list of types. • Commercial usability services − Third-party laboratory evaluation − Third-party usability inspection • Remote questionnaire or survey • Collaborative remote evaluation • Video-conferencing-supported evaluation • Instrumented or automated data collection for remote evaluation Remote Usability Evaluation at a Glance 3 • User-reported critical incident method Some remote evaluation situations call for a portable usability evaluation unit, by means of which the laboratory is taken to users in their normal work environment. Portable units often contain special usability evaluation equipment, including a laptop computer, video, and audio equipment. While this is an area of growing interest, it is outside the scope of this work. COMMERCIAL USABILITY SERVICES A number of commercial usability evaluation services are now available to software developers. Developers use the network to communicate design documents, software samples, and/or prototypes to remote contractual evaluators, but the network is not used to connect to remote users. The evaluation is local to the contractual evaluators and remote from the developers, with results being returned via the network. Contractual evaluation services can be a good bargain for developer groups who have only occasional need for usability evaluation and cannot afford their own facilities. However, the quality of these services can vary, the methods can be ad hoc (e.g., intuitive inspection without use of guidelines), and the process is not always suitable for the specific needs of a development group. Two possible variations of local evaluation at remote sites are described below. Third-party laboratory evaluation Formal laboratory-based usability evaluation offered by consulting groups provides usability evaluation with representative users and tasks. Results include quantitative performance measures, user opinions and satisfaction ratings, recommendations for application improvement, and sometimes even a copy of evaluation session videotapes for review by the development team . Third-party usability inspection Some developers send their designs to remote contractors who perform local evaluation using ad hoc, intuitive interface inspection, drawing on design guidelines, user profiles, and software standards. Without empirical observation of users and a more formal process, results can vary depending on the knowledge and skills of the evaluators. As an example, this kind of evaluation is done by Vertical Research, Inc. (1997) for Microsoft Windows-based products. REMOTE QUESTIONNAIRE OR SURVEY A software application can be augmented to display a user questionnaire to gather subjective user preference data about the application and its interface. Appearance of the questionnaire, requesting feedback related to usability, is triggered by an event (including task completion) during usage. Responses are batched and sent to the developers. As an example, the User Partnering (UP) Module  from UP Technology (Abelow, 1993) uses event-driven triggers to “awaken” dialogues that ask users questions about their usage. Approaches based on remote questionnaires have the advantage that they capture remote user reactions while they are fresh, but they are limited to subjective data based on questions pre-written by Remote Usability Evaluation at a Glance 4 developers or evaluators. Thus, many of the qualitative data normally acquired in laboratory evaluation (i.e., the data directly useful in identifying specific usability problems) are lost. COLLABORATIVE REMOTE EVALUATION The distributed location of users and evaluators yields the opportunity for collaborative usability evaluation via the network (Hammontree, Weiler, and Nayak, 1994). Evaluators (at a usability laboratory) and remote users (at their natural work setting) are connected through the Internet and/or a dial-up telephone line, and use commercially available software (e.g., Microsoft  NetMeeting, Farallon Timbuktu Pro) to assist them during remote evaluation of the user interface, supporting both synchronous and asynchronous collaboration. Typical tools available in such applications include: • real-time application sharing (e.g., the capability to open a word processor at the user’s computer and share it with the evaluator, even when the evaluator does not have that application installed, such that the user and evaluator take turns to modify a document), • audio conferencing support for user and evaluators to talk during the evaluation session (e.g., audio could be sent via an Internet telephone tool or regular telephone), • shared whiteboard with integrated drawing tool for viewing and editing graphic images in real time (e.g., pointing out specific areas by using a remote pointer or highlighting tool, or taking a "snapshot" of a window and then pasting the graphic on whiteboard surface), and/or • file transfer capabilities (e.g., the capability of a user to transfer a document including snapshots of the user interface being evaluated to the evaluators’ computer). TeamWave Workplace, an example of a collaborative tool that could be used for remote evaluation, includes a shared space or virtual room for a work group. As in the case of real-life team meeting rooms, a virtual room may contain several objects (e.g., documents) relevant to the team's work. The user and evaluator can be present in the room at the same time, so they are working together synchronously in real-time, or they may be working in the room at different times, so the user can leave things in that room for the evaluator to analyze asynchronously. This type of asynchronous collaboration gives the evaluator the flexibility to do the evaluation all at once or at a time that is convenient. VIDEO-CONFERENCING-SUPPORTED EVALUATION When the user and evaluator are in adjacent rooms, local usability evaluation requires a video/audio cable (or wireless communication) between the rooms. Users located more remotely can be connected to evaluators using the network and video conferencing software, as an extension of this video/audio cable (Hartson et al., 1996). This kind of remote evaluation, using video teleconferencing over the network as a mechanism to transport video data in real time, perhaps comes the closest to the effect of local evaluation. Currently, the primary obstacle to this approach is the limited bandwidth of the network, occasioning communication delays and low video frame rates. INSTRUMENTED OR AUTOMATED DATA COLLECTION FOR REMOTE EVALUATION An application and its interface can be instrumented with embedded metering code to collect and return a journal or log of data occurring as a natural result of usage in users’ normal working environments. Data Remote Usability Evaluation at a Glance 5 captured by such applications (e.g., WinWhatWhere  WinWhatWhere) represent various user actions made during task performance, are often very detailed, and include logging of: • program usage (e.g., which programs users utilize most frequently), • project time (e.g., time spent working on specific application), • Internet usage (e.g., monitor use of on-line time), • comments to the system, • keystrokes and mouse movements, and • any other activity, producing custom-built reports. The logs or journals of data are later analyzed using pattern recognition techniques (Siochi and Ehrich, 1991) to deduce where the usability problems have occurred. This approach has the advantage of not interfering at all with work activities of the user and can provide automated usability evaluation for certain kinds of usability problems. However, for formative evaluation it can be difficult to infer some types of usability problems effectively. This method has been used successfully for summative evaluation, marketing trials, and beta evaluation. USER-REPORTED CRITICAL INCIDENT METHOD This method applies selective data collection triggered directly by users while performing tasks in their normal work context (Hartson et al., 1996). Users are trained to identify critical incidents and report specific information about these events (e.g., description and severity of problem). The reports are transmitted to developers along with context information about the user task (i.e., what user was trying to do when problem occurred) and the system itself (e.g., name and/or location of screen where problem occurred), as well as clips containing screen sequence actions. Evaluators use these data, approximating the qualitative data normally taken in the usability laboratory, to produce usability problem descriptions. The user-reported critical incident method for remote evaluation, which is the subject of the work reported here, has potential for cost-effectiveness, since the user gathers the data and evaluators look (at least in theory) only at data that relate to usability problems. CLASSIFICATION OF REMOTE EVALUATION METHODS Remote usability evaluation encompasses several methods for evaluating user interfaces at a distant location. Since it is useful to understand how each method works under different situations, this section presents a classification that distinguishes among the characteristics of each remote evaluation method. The classifications made here are not absolute, but are relative and representative, and are based on insight gained during this study, offered to promote an intuitive comparison of the methods. The following attributes of remote usability evaluation methods are considered: • type of users involved, • time of evaluation, • user location during evaluation, Remote Usability Evaluation at a Glance 6 • person or role who identifies critical incidents and/or problems during task performance, • type of tasks (user’s own tasks or tasks predefined by evaluator), • level of interaction between user and evaluator, • type of data gathered, • type of equipment used for collecting data (e.g., videotape), • cost to collect data, • cost to analyze data and create usability problem descriptions, and • quality or usefulness of collected data. Laboratory-based usability evaluation is included in each classification as a benchmark method for comparison with the remote evaluation methods. TYPES OF USERS INVOLVED The diagram in Figure 2 characterizes typical users involved in each type of remote evaluation described in the previous section. In the user-reported critical incident method, remote questionnaire, and instrumented or automated data collection, only real users participate in remote evaluation, as seen in the left of Figure 2. Collaborative evaluation, video-conferencing-supported evaluation, and third-party laboratory evaluation (center of the figure) can use both real and/or representative users during remote evaluation. Finally, third-party usability inspection is the only method that does not involve users during evaluation (evaluator performs intuitive or heuristic inspection of the user interface). Third-party usability inspection No Users Involved REAL USERS REPRESENTATIVE USERS BOTH Instrumented or automated data collection User-reported critical incident method Remote questionnaire or survey Collaborative remote evaluation Video-conferencingsupported evaluation Traditional lab-based usability evaluation Third-party laboratory evaluation None Figure 2. Characterization of types of users typically involved in remote evaluation TIME AND USER LOCATION Johansen (1988) distinguished several different situations of group work along the dimensions of time and place. Adapting Johansen’s work, Figure 3 presents different situations that can occur during task performance with remote evaluation along the dimensions of time and user location (place). Remote Usability Evaluation at a Glance 7 Instrumented or automated data collection User-reported critical incident method Remote questionnaire or survey Collaborative remote evaluation Collaborative remote evaluation Video-conferencingsupported evaluation Different (Asynchronous) Same (Synchronous) TIME OF EVALUATION U se r’ s ow n w or ki ng